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Abstract 

In  collaborative  augmented  reality  (AR)  missions,  much  georegis¬ 
tered  information  is  collected  and  sent  to  a  command  and  control 
center.  This  paper  describes  the  concept  and  prototypical  imple¬ 
mentation  of  a  mixed  reality  (MR)  based  system  that  integrates 
georegistered  information  from  AR  systems  and  other  sources  on 
a  virtual  globe.  The  application  can  be  used  for  a  command  and 
control  center  to  monitor  the  field  operation  where  multiple  AR 
users  are  engaging  in  a  collaborative  mission.  Google  Earth  is  used 
to  demonstrate  the  system,  which  integrates  georegistered  icons, 
live  video  streams  from  field  operators  or  surveillance  cameras,  3D 
models,  and  satellite  or  aerial  photos  into  one  MR  environment. 

Index  Terms:  H.5.1  [Information  Interfaces  and  Presentation]: 
Multimedia  Information  Systems — Artificial,  augmented,  and  vir¬ 
tual  realities;  H.5.3  [Information  Interfaces  and  Presentation]: 
Group  and  Organization  Interfaces  — Computer- supported  cooper¬ 
ative  work 

1  Introduction 

Augmented  reality  (AR)  and  mixed  reality  (MR)  are  being  used 
in  urban  leader  tactical  response,  awareness  and  visualization  pro¬ 
totypes  [3].  Fixed-position  surveillance  cameras,  mobile  cameras, 
and  other  image  sensors  are  widely  used  in  security  monitoring  and 
command  and  control  operations.  The  ability  to  let  the  command 
and  control  center  have  a  view  of  what  is  happening  on  the  ground 
in  real  time  is  very  important  for  situation  awareness.  Decisions 
need  to  be  made  quickly  based  on  a  large  amount  of  information 
from  multiple  image  sensors  from  different  locations  and  angles. 
The  users  must  understand  the  relationship  among  the  images,  and 
recreate  a  3D  scene  in  their  minds.  It  is  a  frustrating  process,  espe¬ 
cially  when  it  is  an  unfamiliar  area,  as  may  be  the  case  for  tactical 
operations. 

Our  objective  is  to  integrate  geometric  information,  georegis¬ 
tered  image  information,  and  other  georeferenced  information  into 
an  MR  environment  that  reveals  the  geometric  relationship  among 
them.  The  system  can  be  used  for  security  monitoring,  or  by  a  com¬ 
mand  and  control  center  to  direct  a  field  operation  in  an  area  where 
multiple  operators  are  engaging  in  a  collaborative  mission. 

On  the  reality-virtuality  continuum,  our  work  is  close  to  aug¬ 
mented  virtuality,  where  the  real  world  images  are  dynamically  in¬ 
tegrated  into  the  virtual  world  in  real  time  [4],  This  project  works 
together  closely  with  our  AR  situation  awareness  application,  so  we 
will  refer  it  as  an  MR  based  application  in  this  paper. 

Although  projecting  real  time  images  on  top  of  3D  models  has 
been  widely  practiced  [1,  2],  and  there  are  some  attempts  on  aug¬ 
menting  live  video  streams  for  remote  participation  [6]  and  remote 
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videoconferencing  [5],  no  work  has  been  done  on  integrating  geo¬ 
registered  information  on  a  virtual  globe  for  MR  applications. 

The  novelty  of  our  approach  lies  in  overlaying  georegistered  in¬ 
formation,  such  as  real  time  images,  icons,  and  3D  models,  on  top 
of  Google  Earth.  This  not  only  allows  a  viewer  to  view  it  from  the 
camera’s  position,  but  also  a  third  person  perspective.  When  in¬ 
formation  from  multiple  sources  are  integrated,  it  provides  a  useful 
tool  for  command  and  control  centers. 

2  Methods 

Our  approach  is  to  partially  recreate  and  update  the  live  3D  scene 
of  the  area  of  interest  by  integrating  information  with  spatial  geo¬ 
registration  and  time  registration  from  different  sources  on  a  virtual 
globe  in  real  time  that  can  be  viewed  from  any  perspective.  This  in¬ 
formation  includes  video  images  (fixed  or  mobile  surveillance  cam¬ 
eras,  traffic  control  cameras,  and  other  video  cameras  that  are  ac¬ 
cessible  on  the  network),  photos  from  high  altitude  sensors  (satellite 
and  unmanned  aerial  vehicle),  tracked  objects  (personal  and  vehicle 
agents  and  tracked  targets),  and  3D  models  of  the  monitored  area. 

We  use  GIS  or  virtual  globe  systems  as  platforms  for  such  a  pur¬ 
pose.  The  freely  available  virtual  globe  application,  Google  Earth, 
is  very  suitable  for  such  an  application,  and  was  used  in  our  prelim¬ 
inary  study  to  demonstrate  the  concept. 

2.1  Projection 


Figure  1 :  Video  images  of  a  parking  lot  and  part  of  a  building  from 
a  surveillance  video  camera  on  the  roof  top  (left)  and  an  AR  user  on 
the  ground  (right). 

In  general  there  are  two  kinds  of  georegistered  objects  that  need 
to  be  displayed  on  the  virtual  globe.  One  is  objects  with  3D  posi¬ 
tion  information,  such  as  icons  representing  the  position  of  users  or 
objects.  The  other  is  2D  imagery  (Figure  1). 

To  overlay  iconic  georegistered  information  on  Google  Earth  is 
relatively  simple.  Overlaying  the  2D  live  video  images  on  the  vir¬ 
tual  globe  is  complex.  The  images  need  to  be  projected  on  the 
ground,  as  well  as  on  all  the  other  objects,  such  as  buildings  (Fig¬ 
ure  2).  From  a  strict  viewpoint  these  projections  couldn’t  be  per¬ 
formed  if  not  all  of  the  3D  information  were  known  along  the  pro¬ 
jection  paths.  However,  it  is  accurate  enough  in  practice  to  just 
project  the  images  on  the  ground  and  the  large  objects  such  as  build¬ 
ings.  Since  it  is  difficult  to  recreate  3D  models  in  real  time  with  few 
images,  we  are  projecting  the  images  on  known  3D  models  instead 
at  least  in  the  early  stages  of  the  study. 
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Figure  2:  Recreated  scene  viewed  with  buildings  on  Google  Earth. 


To  display  the  images  on  Google  Earth  correctly,  we  need  to  cre¬ 
ate  the  projected  texture  maps  on  the  ground  and  the  buildings.  This 
requires  the  projected  images  and  location  and  orientation  of  the 
texture  maps.  We  used  an  OpenSceneGraph  based  rendering  pro¬ 
gram  to  create  the  texture  maps  in  the  frame-buffer.  This  is  done  by 
treating  the  video  image  as  a  rectangle  with  texture.  The  rectangle’s 
position  and  orientation  are  calculated  from  the  camera’s  position 
and  orientation.  When  viewing  from  the  camera  position  and  using 
proper  viewing  and  projection  transformations,  the  needed  texture 
maps  can  be  created  by  rendering  the  scene  to  the  frame-buffer. 

To  create  the  texture  map  of  the  wall,  an  asymmetric  perspective 
viewing  volume  is  needed.  The  viewing  direction  is  perpendicular 
to  the  wall.  The  viewing  volume  is  a  frustum  which  is  formed  with 
the  camera  position  as  the  apex,  and  the  wall  as  the  base. 

When  projecting  on  the  ground,  we  first  divide  the  area  of  inter¬ 
est  into  grids  of  proper  size.  When  each  rectangular  region  of  the 
grid  is  used  instead  of  the  wall,  the  same  projection  method  for  the 
wall  described  above  can  be  used  to  render  the  texture  map  in  the 
frame-buffer. 

The  zoom  factor  of  the  video  camera  can  be  converted  to  the  field 
of  view.  Together  with  the  position  and  orientation  of  the  camera 
that  are  tracked  by  GPS,  inertial  devices,  and  pan-tilt  readings  from 
the  camera,  we  can  calculate  where  to  put  the  video  images.  The 
position  and  size  of  the  image  can  be  arbitrary  as  long  as  it  is  along 
the  camera  viewing  direction,  with  the  right  orientation  and  a  pro¬ 
portional  size. 

2.2  Rendering 

The  rendering  of  the  texture  is  done  with  our  AR/MR  rendering 
engine  which  is  based  on  OpenSceneGraph.  A  two-pass  rendering 
process  is  performed  to  render  the  shadows  of  the  buildings. 

In  the  first  pass,  all  of  the  3D  objects  in  our  database  are  disabled 
and  only  the  camera  image  rectangle  is  in  the  scene.  The  rendered 
image  is  grabbed  from  the  frame-buffer.  Thus  a  projected  image  of 
the  video  is  obtained.  In  the  second  pass  the  camera  image  rectangle 
is  removed  from  the  scene.  The  grabbed  image  in  the  first  pass 
is  used  as  a  texture  map  and  applied  on  the  projection  plane  (the 
ground  or  the  walls).  All  the  3D  objects  in  the  database  (mainly 
buildings)  are  rendered  as  solid  surfaces  with  a  predefined  color  so 
that  the  shadows  are  created.  The  resulting  image  is  read  from  the 
frame-buffer  and  used  as  a  texture  map  in  Google  Earth.  A  post¬ 
processing  stage  changes  the  blocked  area  to  transparent  so  that  the 
satellite/aerial  photos  on  Google  Earth  are  still  visible. 

3  Results 

We  implemented  an  information  integration  prototype  module  with 
the  Battlefield  Augmented  Reality  System  (BARS)  [3].  This  mod¬ 
ule  is  an  HTTP  server  that  sends  icons  and  image  data  to  Google 


Earth.  The  testing  area  is  a  parking  lot  and  some  buildings  nearby. 
The  left  side  image  in  Figure  1  is  the  video  image  from  a  fixed  pan¬ 
tilt-zoom  network  surveillance  camera  (AXIS  213  PTZ)  mounted 
on  top  of  the  roof  of  a  building  by  the  parking  lot.  This  simulates 
a  forward  observation  post  in  military  applications  or  surveillance 
camera  in  security  applications.  Another  AR  user  is  on  the  ground 
of  the  parking  lot;  the  image  captured  by  this  user  is  shown  on  the 
right  side  of  Figure  1 .  The  command  and  control  center  is  located 
at  a  remote  location  running  the  MR  application  and  Google  Earth. 

The  images  are  projected  on  the  buildings  as  well  as  on  the 
ground  and  overlaid  on  Google  Earth  (Figure  2),  together  with  the 
icon  of  an  AR  user  and  the  icon  representing  the  camera  on  the  roof 
of  the  building.  The  parking  lot  part  is  projected  on  the  ground  and 
the  building  part  (the  windows,  the  door,  and  part  of  the  walls)  is 
projected  on  vertical  polygons  representing  the  walls  of  the  build¬ 
ing.  When  the  texture  was  created,  the  part  that  is  not  covered  by 
the  video  image  is  transparent  so  it  blended  into  the  aerial  image 
well.  The  part  of  the  view  blocked  by  the  building  is  removed  from 
the  projected  image  on  the  ground. 

The  result  shows  the  value  of  this  study  which  integrates  infor¬ 
mation  from  multiple  sources  into  one  mixed  environment.  From 
the  source  images,  it  is  difficult  to  see  how  they  are  related.  By 
integrating  images,  icons,  and  3D  models,  it  is  very  easy  for  the 
command  and  control  center  to  monitor  what  is  happening  live  on 
the  ground.  In  this  particular  position,  the  AR  user  on  the  ground 
and  the  simulated  forward  observation  post  on  the  roof  top  can  not 
see  each  other. 

4  Conclusion 

In  this  preliminary  study,  we  investigated  the  methods  of  integrat¬ 
ing  georegistered  information  on  a  virtual  globe.  The  application 
can  be  used  for  a  command  and  control  center  to  monitor  the  field 
operation  where  multiple  AR  users  are  engaging  in  a  collaborative 
mission.  Google  Earth  is  used  to  demonstrate  the  methods.  The 
system  integrates  georegistered  icons,  live  video  streams  from  field 
operators  or  surveillance  cameras,  3D  models,  and  satellite  or  aerial 
photos  into  one  MR  environment. 

This  work  is  sponsored  by  DARPA  under  the  ULTRA- Vis  Pro¬ 
gram  for  work  as  a  performing  member  of  the  Lockheed  Martin 
team,  contract  #  FA8650-09-C-7908.  The  views,  opinions,  and/or 
findings  contained  in  this  article/presentation  are  those  of  the  au¬ 
thor/presenter  and  should  not  be  interpreted  as  representing  the  of¬ 
ficial  views  or  policies,  either  expressed  or  implied,  of  the  Defense 
Advanced  Research  Projects  Agency  or  the  Department  of  Defense. 
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